posted 12-11-2007 03:27 PM            

The Comparison Question Test: Does It Work and If So How?
Dr.s Heinz Offe, Susanne Offe
Law and Human Behavior, Volume 31, Number 3, June 2007 , pp. 291-303(13)

In a mock crime study of the comparison question test (CQT), 35 subjects decided to participate as guilty and 30 as innocent. Two conditions were varied: Explaining the comparison questions in the pretest interview and re-discussing comparison questions between charts. Higher identification rates (∼90% for guilty and innocent participants) were achieved in groups with explanation of comparison questions than in groups without explanation. Re-discussing comparison questions had no effect on identification rates. Ratings of subjective stress due to relevant and comparison questions were also obtained and can be seen as indicators of the significance of the questions. The significance of comparison questions was hardly affected by the different testing conditions. When effects are detectable at all, they contradict theoretical expectations in their direction. Results are discussed in terms of the significance of comparison questions used in polygraph testing.

Here, we have yet another peer reviewed scientific study, this time published in 2007 from researchers in Germany, that demonstrates the value of the comparison question polygraph test. The authors title question says it all. Does it work and if so how? Their methods were some of the best I have seen in over 20 years as a polygrapher The usual arguments about mock crime studies center around the argument that mock crime studies don’t replicate “real life” and that people who are “programmed” as guilty or innocent don’t display the same emotions or reactions as “real life” people. Well these researchers did a number of things to quell that argument.

First they let the participants chose whether they wanted to be the criminals or the innocent suspects. After all that’s what people in “real life” do. They then gave the guilty subjects two weeks to go about committing the crime, thereby allowing them to chose the most appropriate time to do the deed unobserved. The guilty subjects had to steal a money voucher from a box on a desk in an office area accessible to all the participants. This again replicates the manner in which a “real crime” would occur.

Next, they conducted the polygraph examinations between one week and five months of the “crime” again replicating “real life” in much the same way one might become a suspect in a criminal case at some point during the investigation.

Their questions to be answered were:

1. Is it possible to achieve significantly higher than chance identification rates in a mock crime study under more realistic conditions of subjects deciding for themselves, whether they participate as guilty or innocent?
2. Does explanation of the comparison questions (CQ) in the pre-test interview affect the identification rates as expected?
3. Does discussion of the comparison questions between tests – based on the concept of “delicate balance” affect identification rates?
4. Is the subjectively felt stress imposed by questions related to test results based on physiological measures?
5. does the subjective uncertainty about the truthfulness of negating the CQ contribute to the significance of these questions.

Rewards were offered to motivate the participants. The guilty were offered double the reward should they be later identified as non-deceptive and half for those innocent participants who successfully pass the polygraph.

The results were very telling. The subjects were divided into 4 groups with Group 1 receiving an explanation of the CQ both during the pre-test and between test charts. Group 2 received explanation of the CQ only during the pre-test interview. Group 3 received no CQ explanation during the pre-test but did receive discussion of the questions between test charts. Group 4 received neither explanation of the CQ during the pre-test nor discussion of those questions between test charts.

When explanation of the CQ was accomplished during the pre-test and between test charts, the results were 88.9% for both conditions being correctly identified. When explanation of the CQ was accomplished during the pre-test but with no discussion between test charts, the identification rates were 93.3%. Group 3 participants who received only between chart discussion of the CQ were identified correctly at a rate of 80.0% and Group 4 participants who received no explanation or discussion were correctly identified by condition at a rate of 58.8%

Questionnaires completed by all participants after their polygraph examinations, revealed that both guilty and innocent participants subjectively “felt” that the relevant questions were the most important of the questions answered on the test. The guilty, however “felt” that they reacted stronger to the relevant questions and the innocent “felt” they responded stronger to the comparison questions.

This test has confirmed, yet again that when performing the CQT under normal conditions in mock crime studies rates of 90% or better can be obtained even when the participants chose their condition. Innocent subjects subjectively felt the relevant questions were the most important yet they felt they responded stronger to the CQ. Guilty subjects actually were identified at lower rates when the explanation of the CQ was not given and innocent subjects were correctly classified at a higher rate when explanation of the CQ was provided. In other words, it helped them pass the test at higher rates.

To answer Mr. Maschke’s obvious questions:

1. Yes the examiners were blind to conditions of the participants.
2. Even with inconclusive results included in the study, the resulting identification rates differed only slightly and were not significant.
3. The “delicate balance” argument did not play out. Discussion of the CQ between the test charts did not increase or decrease identification rates.

posted 12-11-2007 03:43 PM            

Thank you Skip
E

posted 12-11-2007 04:06 PM            

Prediction: no effect. Look at Lethe. He still believes knowledge of the test will clearly result in poorer performance. Research means nothing to them unless it can be twisted to support their cause.

posted 12-11-2007 04:15 PM            

Then maybe have someone read the article who is intelligent but somewhat naive concerning polygraph to see if they can comprehend the significance of the findings.

These aren't criticisms just suggestions I think you could post it as is and have the desired effect. Great article

------------------
Ex scientia veritas

posted 12-11-2007 06:23 PM            

I was considering suggesting this one for the next round of anti-poly-follies.

posted 12-11-2007 07:21 PM            

It makes me wonder if I have done a good pre-test if all of the CQs are flat

------------------
Ex scientia veritas

posted 12-14-2007 10:49 AM            

I referenced it last night, but I haven't looked to see what's going on - I can only take so much of it.

posted 12-15-2007 07:58 AM            

its 1991 JAP

I'm concerned that GM is twisting this a little.

It's also important to be thoughtful about any specificity deficits that parallel other findings, like Blackwell 1999.

I'll be back on Monday, and get get it from the Library, but it'd be great to read it sooner.

posted 12-15-2007 03:13 PM            

I'm taking a break from icy roads and sub-zero temps.

Does anyone know if the Patrick and Iacono 1991 procedures are consistent with RCMP procedures?

Sorry Patric and Iacono 1991 is a different study than the one GM seems to be referring to. It was interesting though, and still provides some retort.

--------------

The authors did not seem to specify what proportion of the exams were DIR (direct - "did you...") and IND (indirect - "dyk...", "did you plan...", "have you personally received...")

GM criticizes the 50% specificity rate, (which was actually 55%).

This is important, because the Blackwell data, also showed poor specificity (and good sensitivity).

Keep in mind that this is as much about procedures as it is about construct validity.

I don't know about RCMP scoring features/criteria. But the DoDPI criteria in use during the 1990s was basically a "score anything that moves" criteria - regarless of Kircher's discriminate analysis in 1988. Present criteria have been improved.

The advantage of long (score anything) criteria lists, is that it increases the sensitivity of the test to deception, because a broad range of possible reactions attract the examiner's concern. (Any of the criteria will move score in the DI direction). The disadvantage of long and poorly researched criteria/feature lists is a reduction in specificity due to the gauntlet effect - one has to not show reaction to anything to move scores in the NDI direction.

So... Duh. The found poor specificity - so did Blackwell 1999 - for probably the same reason.

Generalizing those early findings to modern situations is unsound, and the studies need to be replicated AGAIN.

Another contributor to the specificity deficiencies observed by P&I 1991 is that the IND tests were evaluated using spot rules while the DIR exams were evaluated using total scores. IND exams were multi-facet. DIR were single-issue.

We know now, thanks to Krapohl and others, that the spot rule can cause increased sensitivity at the cost of decreased specificty. We also know now, from Senter and Dollins 2003, that two stage rules correct for that, and permit increased sensitivity without an excess risk of decreased specificity. This is the exact same concern, as discussed a few days ago, involving the addition-rule and inflated alpha contributing to increased FPs. Senter's two-stage rules are procedurally analagous to the mathematical implementation of a Bonferoni correction to the desired alpha. (I'm quite sure he knows that - he just didn't burden us with statistical concepts and simply told us the solution.)

P&I used -2 and +1 for spot cutscores. and +/-6 for totals. We don't really know the p-values for those spot or total scores. However, we do know from Senter 2005? that MGQT/spot scoring rules provide sensitivity in the ballpark reported by Patrick and Iacono - and they appear to have scored the IND exams using MGQT/spot rules. DIR exams were presumably scored with the spot rule.

GM neglected to point out that original examiners substantially outperformed the blind reviewers - esp with the truthful cases. Consistent with other findings. The chi-square analysis showed hit rates significantly above chance for both decetpive and truthful subjects (p = <.01 and p = <.01), though they pointed out a difference between decisions and scores for the original scorers - with the decisions for truthful subjects, based on scores, not significantly above chance (interesting).

The findings are interesting compared with those of Blackwell 1999 who showed accuracy ~98% and ~60% for deceptive and truthful with ZCT exams and worse for MGQT, ~96 and ~25%. Error rates were ~50 and ~30% for truthful, for MGQT and ZCT exams, along with 0% and 2% for deceptive for MGQT and ZCT exams. INCS were ~26% and 6% for truthful and deceptive for ZCT exams and ~22% and ~3% for MGQT exams. Fliess' Kappa was .57 for both.

We know that the bigger is better rule was not taught until the late 1990s, and the Blackwell cases appear to have been scored during 1996.

Read Krapohl and Norris 2000 for more info on how the spot scoring rule affects observed outcomes.

We have no description of the procedures of the original scorers for the P & I cases, and they may have differed from those of the blind scorers.

P&I investigated only crime-type and age, as mitigating variable for false-positive errors, and made no effort to evaluate decision rules or procedures (as Homer Simpson would say: "Doh!").

They should have investigated procedural and decision rule bias.

Krapohl, Shull, and Ryan investigated the confession criterion and found no significant differences in the scores of deceptive persons who confessed and those who did not confess - suggesting no great differences in how the test works with deceptive confessors compared to deceptive non-confessors. I'm no expert in statistical power analysis but there is nothing obviously deficient about the study.

In short, P & I can easily be understood as a critique of methodology - some of which has been or is being rectified.

What P & I ascribe to criterion bias may actually be attributable to a procedural and decision rule bias.

Results from those studies are difficult to compare to modern results, until someone takes the time to re-evaluate those samples using modern testing principles.

r

ASIDE: nopolyforme is beginning to sometimes remind me of digithead, though with less pretense around the statistical expertise he gleaned from NAS. I'm tempted to go back and re-read their past postings.

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)